Weight Perturbation as a Method for Improving Performance of Deep Neural Networks

Learnable weights in very deep neural networks are not utilized in an efficient way i.e. the variance of weights in some non-performing layers is close to zero. One approach taken to mitigate this problem, has been the connection pruning, in which some of the weights in a network are masked with zeros, without negatively affecting the network performance. The drawback of such approach is that in the GPU implementations of neural networks, pruning the weights does not affect the network training or inference speed. In this paper we propose an alternative solution to the problem of inefficient utilization of network learning capability - perturbing the weights of the network during training with additive, zero-mean noise. It causes the variance of weights in non-performing layers to rise, and can improve the performance of neural networks. In this paper we show the effect of the weight perturbation on the performance of five well known network architectures, while testing them on the task of CIFAR100 classification.